Cinematic mastering for virtual reality and augmented reality

ABSTRACT

An entertainment system provides data to a common screen (e.g., cinema screen) and personal immersive reality devices. For example, a cinematic data distribution server communicates with multiple immersive output devices each configured for providing immersive output (e.g., a virtual reality output) based on a data signal. Each of the multiple immersive output devices is present within eyesight of a common display screen. The server configures the data signal based on digital cinematic master data that includes immersive reality data. The server transmits the data signal to the multiple immersive output devices contemporaneously with each other, and optionally contemporaneously with providing a coordinated audio-video signal for output via the common display screen and shared audio system.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/712,820, filed Dec. 12, 2019, which is a continuation of U.S. patentapplication Ser. No. 15/289,174 filed Oct. 8, 2016 and issued as U.S.Pat. No. 10,511,895, which claims priority pursuant to 35 U.S.C. §119(e) to U.S. Provisional Patent Application Ser. No. 62/239,782 filedOct. 9, 2015, the disclosures of all of which are hereby incorporated byreference, in their entireties.

FIELD

The present disclosure relates to the production, configuration, andproviding, by a computer, of digital data for virtual reality oraugmented reality output.

BACKGROUND

“Virtual reality” is a term that has been used for various types ofcontent that simulates immersion in a three-dimensional (3D) world,including, for example, various video game content, and animated filmcontent. In some types of virtual reality, a user can navigate through asimulation of a 3D environment generated based on the computer model, bycontrolling the position and orientation of a virtual camera thatdefines a viewpoint for a 2D scene that is displayed on atwo-dimensional display screen. A variation of these technologies issometimes called “augmented reality.” In an augmented reality setup, thedisplay technology shows a combination of the user's surroundings thatis “augmented” by one or more digital objects or overlays. Augmentedreality content may be as simple as textual “heads up” information aboutobjects or people visible around the user, or as complex as transformingthe entire appearance of the user's surroundings into a fantasyenvironment that corresponds to the user's real surroundings.

Virtual reality (VR) and augmented reality (AR) have been applied tovarious types of immersive video stereoscopic presentation techniquesincluding, for example, stereoscopic virtual reality headsets. Headsetsand other presentation methods immerse the user in a 3D scene. Lenses inthe headset enable the user to focus on a lightweight split displayscreen mounted in the headset only inches from the user's eyes.Different sides of the split display show right and left stereoscopicviews of video content, while the user's peripheral view is blocked. Inanother type of headset, two separate displays are used to showdifferent images to the user's left eye and right eye respectively. Inanother type of headset, the field of view of the display encompassesthe full field of view of eye including the peripheral view. In anothertype of headset, an image is projected on the user's retina usingcontrollable small lasers, mirrors or lenses. Either way, the headsetenables the user to experience the displayed virtual reality contentmore as if the viewer were immersed in a real scene. In the case ofaugmented reality (AR) content, the viewer may experience the augmentedcontent as if it were a part of, or placed in, an augmented real scene.

These immersive effects may be provided or enhanced by motion sensors inthe headset that detect motion of the user's head and adjust the videodisplay(s) accordingly. By turning his head to the side, the user cansee the virtual reality scene off to the side; by turning his head up ordown, the user can look up or down in the virtual reality scene. Theheadset may also include tracking sensors that detect position of theuser's head and/or body and adjust the video display(s) accordingly. Byleaning or turning, the user can see the virtual reality scene from adifferent point of view. This responsiveness to head movement, headposition and body position greatly enhances the immersive effectachievable by the headset. The user may be provided the impression ofbeing placed inside or “immersed” in the virtual reality scene. As usedherein, “immersive” generally encompasses both VR and AR.

Immersive headsets and other wearable immersive output devices areespecially useful for game play of various types, which involve userexploration of a modelled environment generated by a rendering engine asthe user controls one or more virtual camera(s) using head movement, theposition or orientation of the user's body, head, eye, hands, fingers,feet, or other body parts, and/or other inputs. To provide an immersiveexperience, the user needs to perceive a freedom of movement that is insome way analogous to human visual perception when interacting withreality. Content produced for VR can provide this experience usingtechniques for real-time rendering that have been developed for varioustypes of video games. The content is may be designed as athree-dimensional computer model with defined boundaries and rules forrendering as video output. This content can be enhanced by stereoscopictechniques to provide stereoscopic output, sometime referred to as “3D,”and associated with a VR application that manages the rendering processin response to movement of the VR headset, to produce a resulting VRexperience. The user experience is very much like being placed inside arendered video game.

In other types of VR and AR, the simulated 3D environment may be usedprimarily to tell a story, more like traditional theater or cinema. Inthis type of VR or AR, the added visual effects may enhance the depthand richness of the story's narrative elements or special effects,without giving the user full control (or any control) over the narrativeitself. However, the technology for experiencing anything similar tocinematic content delivered using VR or AR equipment or methods is in avery early stage of development. Actual implementations of technologyare quite limited, and users have thus far been largely or completelyuntouched by VR or AR in their experience of narrative content.

It would be desirable, therefore, to develop new methods and other newtechnologies for mastering cinematic content for VR and AR use, thatovercome these and other limitations of the prior art and enhance theappeal and enjoyment of narrative content for new immersive technologiessuch as VR and AR.

SUMMARY

This summary and the following detailed description should beinterpreted as complementary parts of an integrated disclosure, whichparts may include redundant subject matter and/or supplemental subjectmatter. An omission in either section does not indicate priority orrelative importance of any element described in the integratedapplication. Differences between the sections may include supplementaldisclosures of alternative embodiments, additional details, oralternative descriptions of identical embodiments using differentterminology, as should be apparent from the respective disclosures.

In an aspect of the disclosure, a computer-implemented method includescommunicating, by a cinematic data distribution server over a wirelessnetwork, with multiple immersive output devices each configured forproviding one of an augmented reality (AR) output or a virtual reality(VR) output based on a data signal, wherein each of the multipleimmersive output devices is present within eyesight of a display screen.For example, the multiple immersive output devices may be worn bymoviegoers or home theater users. The method may include configuring thedata signal based on digital cinematic master data that includes atleast one of VR data or AR data, for example in ways as summarized belowand described elsewhere herein. The method may include transmitting thedata signal to the multiple immersive output devices contemporaneously,such that each of the users receives and processes the data and shares acontemporaneous immersive video experience.

In another aspect, the method may include outputting an image based on avideo data portion of the digital cinematic master data on the displayscreen, contemporaneously with the transmitting. The users may therebyenjoy an AR experience in addition to the video on the screen, or ifusing fully VR equipment that obscures the screen, may enjoy a cinematicpresentation that both supplements and duplicates the presentation onthe screen.

For serving AR immersive output devices, the configuring the data signalmay include encoding the AR data for augmenting video data for output onthe display screen and including the AR data with the video data in thedata signal. The AR data may be configured to provide various effects.In an aspect, the AR data, when received by the multiple immersiveoutput devices, continuously extends images on the display screen toareas beyond an outer limit of the display screen, for each personviewing AR output on one of the multiple immersive output devices. Forexample, a person wearing an AR immersive output device may see elementsof the scene that extend upwards, downwards, or sideways beyond theframe. In another, alternative aspect, the AR data, when received by themultiple immersive output devices causes images that do not appear onthe display screen to appear in a non-screen display volume to eachperson viewing AR output on one of the multiple immersive outputdevices. For example, the non-screen object may be caused to appear infront of, above, or below the display screen, or even behind the viewer.These effects may similarly be provided by configuring VR data for a VRoutput device.

The data signal may be configured to provide each user with an“objective” experience, a “subjective” experience, or a mixture ofobjective and subjective experiences. To provide a subjectiveexperience, the cinematic data distribution server configures the datasignal such that the images that do not appear on the display screen(i.e., the images that are visible only using an AR or VR output device)appear in a coordinate system defined relative to the each personviewing AR or VR output on one of the multiple immersive output devices.To provide an objective experience, the cinematic data distributionserver configures the data signal such that the images that do notappear on the display screen appear in the same coordinate systemdefined relative to the display screen, or in other words, a coordinatesystem that is relative to the cinema or home theater and the same forall immersive output devices. To provide a mixed experience thecinematic data distribution server configures the data signal such thatat least one visible object is defined individually relative to eachperson's subjective coordinate system, while at least one other objectis defined in the common coordinate system and is the same for allviewers.

In another aspect, a user may be able to interact with objects depictedin AR or VR output. Accordingly, the method may include providing in theAR data or VR data code enabling each person viewing AR output on one ofthe multiple immersive output devices to interact with at least one ofthe images, causing the AR output or VR output to change. In a relatedaspect, the method may include changing video output shown on thedisplay screen based on the each person's interaction with at least oneof the images. For example, different versions of a scene may beprovided in stored cinematic data, and the version selected at runtimemay be selected based on an aggregate of feedback from the differentusers. In another related aspect, the method may include providing anelectronic record of a license right to digital content to a useraccount data store associated with the each person, based the eachperson's interaction with at least one of the images. For example, byinteracting with an object or character during a cinematic presentationusing an AR or VR device, a viewer may “earn” the right to use theobject, character, or some related object or character in anotherapplication, such as, for example, a video game application or socialmedia application. In some embodiments, the data store may be, or mayinclude, a computer memory in an AR or VR output device worn by the eachperson.

In another aspect, the method may include provided in the AR data codefor managing a transition between AR output produced from the AR data byan AR output device and video output shown on the display screen, basedat least in part on a geometry of the display screen and each personviewing AR output on one of the multiple immersive output devices.Accordingly, a viewer wearing an AR device may be able to view images onthe display screen and images output by the AR device simultaneously,without any distracting break or disruption between the display screenimage and surrounding AR images.

In another aspect, the method may include providing the VR data foraugmenting video data for output on the display screen, for example byplaying the video data and the VR data at different, non-overlappingtimes, such that ones of the multiple individuals wearing a VR outputdevice view content on the display screen and on a VR output device atdifferent times. The video data and the VR data may also be configuredto play contemporaneously. The video data may be, or may include,two-dimensional non-stereoscopic frame data, stereoscopic frame data, orboth.

In another aspect, the method may include providing haptic control datain the digital cinematic master data, where the haptic control dataactivates a haptic device in coordination with the at least one ofaugmented AR data or VR data.

In a related method, a computer or network of connected computers may beused for configuring digital cinematic master data that includes atleast one of AR data or VR data for providing one of an AR output or aVR output comprising a cinematic presentation; and recording the digitalcinematic master data in a non-transitory computer-readable medium. Theconfiguring may include operations for providing the output effectssummarized above. For example, the configuring may include: preparingthe AR or VR data for augmenting video data configured for projection oroutput on a display screen arranged for viewing by multiple individuals;preparing the AR or VR data for continuously extending images on thedisplay screen to areas beyond an outer limit of the display screen;preparing the AR or VR data for causing images that do not appear on thedisplay screen to appear between the display screen and any one of themultiple individuals wearing an immersive output device (or above,below, behind); or preparing the AR or VR data so that the imagesprovide a subjective experience, an objective experience, or a mixedsubjective and objective experience.

The method may include providing executable instructions (“code”) in theAR or VR data for enabling a user to interact with at least one of theimages, causing the AR output to change in response to user input. Themethod may include configuring the code so that a user's interactionwith at least one of the images causes the video output shown on thedisplay screen to change. The method may include configuring the code sothat a user's interaction with at least one of the images causes anelectronic record of a license right to digital content to be providedto a user account data store associated with the user.

Any of the foregoing methods may be implemented in any suitableprogrammable computing apparatus, by provided program instructions in anon-transitory computer-readable medium that, when executed by acomputer processor, cause the apparatus to perform the describedoperations. An apparatus may include a computer or set of connectedcomputers that is used in video production or is installed in a cinemaor home theater. Other elements of the apparatus may include, forexample, a display screen, an audio output device, and a user inputdevice, which participate in the execution of the method. An apparatusmay include a virtual reality device, such as a headset or other displaythat reacts to movements of a user's head or body to provide theimpression of being placed inside of the rendered scene in which thegame is played.

To the accomplishment of the foregoing and related ends, one or moreexamples comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative aspectsand are indicative of but a few of the various ways in which theprinciples of the examples may be employed. Other advantages and novelfeatures will become apparent from the following detailed descriptionwhen considered in conjunction with the drawings and the disclosedexamples, which encompass all such aspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, nature, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify like elements correspondingly throughout thespecification and drawings.

FIG. 1 is a schematic block diagram illustrating aspects of a system andapparatus for the production and configuration of digital data forvirtual reality or augmented reality output coupled to a distributionsystem.

FIG. 2 is a schematic block diagram illustrating more detailed aspectsof an apparatus for outputting virtual reality or augmented realitycontent.

FIG. 3 is a schematic diagram illustrating aspects of viewingcoordinated immersive and non-immersive content from the perspective ofdifferent viewers.

FIG. 4 is a concept diagram illustrating elements of a system foroutputting immersive content to multiple users in a cinema or hometheater setting.

FIG. 5A is a block diagram illustrating aspects of a media packageholding audio-video data with a predetermined narrative with additionalcontent coordinated with the predefined narrative and configured forproviding an alternative output.

FIG. 5B is a block diagram illustrating aspects of a content displaysystem including coordinated output of immersive and non-immersivecontent.

FIG. 6 is a schematic diagram illustrating components of a stereoscopicdisplay device for providing an immersive VR experience.

FIG. 7 is a diagram illustrating components of, and concepts concerning,a cinema or home theater space for multi-user VR or AR.

FIG. 8A is a flow chart illustrating elements of serving VR or AR datato an AR or VR output device providing a cinema experience.

FIG. 8B is a concept diagram illustrating elements of a system forcoordinating immersive content provided to multiple users in a cinema orhome theater setting.

FIG. 9 is a flow chart illustrating elements of serving VR or AR data toa user of a cinema or home theater non-contemporaneously withtwo-dimensional (2D) data for display on a screen or the like.

FIG. 10 is a flow chart illustrating a method for serving VR or AR datato an AR or VR output device providing a cinema experience.

FIGS. 11-13 are flow charts illustrating further optional aspects oroperations of the method diagrammed in FIG. 10.

FIG. 14 is a conceptual block diagram illustrating components of anapparatus or system for serving VR or AR data to an AR or VR outputdevice providing a cinema experience.

FIG. 15 is a flow chart illustrating a method for configuring digitalcinematic master data for a cinematic presentation.

FIGS. 16 and 17 are flow charts illustrating further optional aspects oroperations of the method diagrammed in FIG. 15.

FIG. 18 is a conceptual block diagram illustrating components of anapparatus or system for configuring digital cinematic master data for acinematic presentation.

DETAILED DESCRIPTION

Various aspects are now described with reference to the drawings. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofone or more aspects. It may be evident, however, that the variousaspects may be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate describing these aspects.

An illustrative system 100 for production and distribution of immersivecontent (e.g., AR and VR) in coordination with non-immersive content(e.g., 2D video with audio, stereoscopic 3D video with audio,non-immersive video games) is shown in FIG. 1. The system 100 mayinclude a set 102 of production activities that produce assets that areshared and used in different ways across related different versions(immersive and non-immersive) of underlying creative content. Creativecontent includes, for example, video data collected by various camerasystems 112, 112, audio data collected and/or generated by audiosubsystems (not shown), and computer modeling/animation data created andarranged from various modeling/animation subsystems 108, 110. Creativecontent may be stored in a data store 106. It should be appreciated thatthe system may include several different data stores (not shown). Aproduction server component 104, which may comprise a family ofproduction applications operating over a computer network, may accessdata in the data store 106 under control of various production staffcontrolling the production process via multiple access terminals 118,116. The number of components shown in system 100 is merelyillustrative. It should be appreciated that a typical feature film orother studio production system will typically include a much largernumber of components than illustrated. Creative and technical directorsoversee the assembly of creative content from the various data sources,configured for immersive output devices and more traditionalnon-immersive devices.

Digital content produced by the system may include various versions ofthe same story, for example, a 2D theater version; a 2D home theaterversion; a mobile device version; stereoscopic 3D version for one ormore of theater, home or mobile devices, a VR version for an in-theaterexperience, optionally in conjunction with supporting 2D or stereoscopic3D content, a VR version for home use, likewise optionally for use withnon-immersive content; an AR version for supplementing non-immersivecontent in a theater, an AR version for supplementing non-immersivecontent in a home theater environment or in a mobile device format, andvideo game content in one or more of the foregoing output formats.Finished productions in each of the various versions may be provided toa home distribution server 120 which may store the different versions ina content data store (not shown) in association with metadata formanaging use and distribution. A least one set of consumers may receivemultiple versions of immersive and non-immersive content in a singledigital content (media) package, whether stored under control of anetwork served 120, or locally on a computer-readable medium such as anoptical disc or memory device.

Different distribution channels each assigned its own server resourcesmay be used to provide content to different sets of end users. Forexample, a cinema distribution server 130 may distribute immersive andconventional content to cinemas for public performance. For illustrativeclarity, one cinema 140 of potentially many cinemas is diagrammed. Eachcinema 140 may include its own server 134 used to distribute digitalcontent to one or more theaters each hosting a performance. Each theater(or the theater, if only a single theater is served by the server 143)includes a cinema screen 136 and one or more viewers each wearing animmersive content consumption device, 132, 138, for example, a VR visoror AR headset. The same underlying audio-video program may thereby bedistributed in different versions for home and cinema use. Both home andcinema versions may include technical elements that coordinate differentimmersive devices contemporaneously playing the audio-video program inan immersive format. In addition, both versions may include elementsthat coordinate play of immersive content with contemporaneous ornon-contemporaneous content playing

In some embodiments, a media package holding coordinated immersive andnon-immersive content may be, or may include, a single computer-readablemedium (for example, an optical disc medium or FLASH memory device) inwhich packaged digital content is stored together. Distribution of anon-transitory, tangible and portable storage medium may reduce networkbandwidth demands and ensure reliable and seamless access to densedigital content by the consumption device. In some embodiments, rapiddistribution to tangible media may be accomplished by distribution fromselected kiosks holding electronic copies of digital content for writingto digital copies. In an alternative, such kiosks may take advantage ofhigh-bandwidth connections to obtain the electronic content fordistribution. In other embodiments, including for example for cinemadistribution, the electronic content may be transmitted over acommunications network and/or computer network and stored directly on amemory device or medium connected to or integrated with a client devicethat will participate in playback of the received content.

Referring to FIG. 2, aspects of a content consumption device 200 forconsuming VR or AR content are illustrated. Several viewers of a hometheater or cinema presentation may be equipped with the contentconsumption device. The apparatus 200 may include, for example, aprocessor 202, for example a central processing unit based on 80x86architecture as designed by Intel™ or AMD™, a system-on-a-chip asdesigned by ARM™, or any other suitable microprocessor. The processor202 may be communicatively coupled to auxiliary devices or modules ofthe 3D environment apparatus 200, using a bus or other coupling.Optionally, the processor 202 and some or all of its coupled auxiliarydevices or modules (examples of which are depicted at 204-216) may behoused within or coupled to a housing 218, for example, a housing havinga form factor of a personal computer, gaming console, smart phone,notepad computer, laptop computer, set-top box, wearable googles,glasses, or visors, or other form factor.

A user interface device 204 may be coupled to the processor 202 forproviding user control input to an immersive content display processoperated by a VR or AR immersive display engine executing on theprocessor 202. User control input may include, for example, selectionsfrom a graphical user interface or other input (e.g., textual ordirectional commands) generated via a touch screen, keyboard, pointingdevice (e.g., game controller), microphone, motion sensor, camera, orsome combination of these or other input devices. Control input may alsobe provided via a sensor 206 coupled to the processor 202. A sensor maycomprise, for example, a motion sensor (e.g., an accelerometer), aposition sensor, a temperature sensor, a location sensor (for example, aGlobal Positioning System (GPS) receiver and controller), aneye-tracking sensor, or a microphone. The sensor 206 may detect a motionor other state of a user interface display, for example, motion of avirtual-reality headset, or the bodily state of the user, for example,skin temperature or pulse.

The device 200 may optionally include an input/output port 208 coupledto the processor 202, to enable communication between a VR/AR engine anda computer network, for example a cinema content server or home theaterserver. Such communication may be used, for example, to enablemultiplayer VR or AR experiences, including but not limited to sharedimmersive experiencing of cinematic content. The system may also be usedfor non-cinematic multi-user applications, for example socialnetworking, group entertainment experiences, instructional environments,video gaming, and so forth.

A display 220 may be coupled to the processor 202, for example via agraphics processing unit (not shown) integrated in the processor 202 orin a separate chip. The display 210 may include, for example, a flatscreen color liquid crystal (LCD) display illuminated by light-emittingdiodes (LEDs) or other lamps, a projector driven by an LCD display or bya digital light processing (DLP) unit, a laser projector, or otherdigital display device. The display device 210 may be incorporated intoa virtual reality headset or other immersive display system. Videooutput driven by a VR/AR immersive display engine operating on theprocessor 202, or other application for coordinating user inputs with animmersive content display and/or generating the display, may be providedto the display device 210 and output as a video display to the user(also referred to herein as the “player”). Similarly, anamplifier/speaker or other audio output transducer 222 may be coupled tothe processor 202 via an audio processing system. Audio outputcorrelated to the video output and generated by the VR/AR display engineor other application may be provided to the audio transducer 222 andoutput as audible sound to the user.

The 3D environment apparatus 200 may further include a random accessmemory (RAM) 214 holding program instructions and data for rapidexecution or processing by the processor during controlling a 3Denvironment. When the device 200 is powered off or in an inactive state,program instructions and data may be stored in a long-term memory, forexample, a non-volatile magnetic, optical, or electronic memory storagedevice 216. Either or both of the RAM 214 or the storage device 216 maycomprise a non-transitory computer-readable medium holding programinstructions, that when executed by the processor 202, cause the device200 to perform a method or operations as described herein. Programinstructions may be written in any suitable high-level language, forexample, C, C++, C#, or Java™, and compiled to produce machine-languagecode for execution by the processor. Program instructions may be groupedinto functional modules, to facilitate coding efficiency andcomprehensibility. It should be appreciated that such modules, even ifdiscernable as divisions or grouping in source code, are not necessarilydistinguishable as separate code blocks in machine-level coding. Codebundles directed toward a specific type of function may be considered tocomprise a module, regardless of whether or not machine code on thebundle can be executed independently of other machine code. In otherwords, the modules may be high-level modules only.

Immersive content may be played with non-immersive content in cinema andhome theater settings, to augment conventional content that is displayedon a 2D screen and viewed using no equipment at all or stereoscopicviewing glasses. FIG. 3 illustrates aspects of augmented content usingAR or VR in a viewing space 300 shared by multiple users 314, 316 and318. A first user 314 wearing an AR headset views a content object 310(“tree”) partially on the screen 302 and partially in an augmented viewvolume 304 surrounding the screen 302. A second user 316 is viewing thescreen 302 with “naked eyes” and no equipment, and sees only the partialview of the object 310 as depicted on the screen 302. The second user316 sees nothing except the actual physical surroundings (e.g., a movietheater or room) in the area surrounding the screen 302.

A third user 318 using a VR headset does not see the screen 302 at all.Instead, the third user sees an equivalent content object 312 for theobject 310 displayed in the screen 302. The object 312 may be located inthe user's VR space 306 in a position relative to the user 318 that isclosely equivalent to the position of the object 310 relative to theusers 314, 316 (i.e., subjectively placed). Hence, all users 314, 316and 318 may share the experience of at least the content playing on thescreen 302, while users 314 and 318 equipped with AR or VR outputdevices can enjoy enhanced content at the same time that non-equippeduser 316 is matching content on the screen 302 only. Both theAR-equipped user 314 and the VR-equipped user 318 may view objects thatappear in front of, above, below, or to a side of the screen. Forexample, the AR-equipped user 314 may see the dragon 320, while theVR-equipped user 318 sees an equivalent dragon 322. Each user sees theimmersive-only objects 320, 322 in their own frame of reference, in theillustrated example, referred to herein as subjective display.

Coordinating output on a 2D screen with VR output may seem unnecessarilyduplicative, but does provide several benefits which are not immediatelyapparent. For example, VR-equipped or AR-equipped audience members andnon-equipped audience members can share a viewing experience together,increasing opportunities for social interaction before, during and afterthe shared experience. In addition, members of the audience who find theVR experience too emotionally intense or experience unpleasantsensations such as vertigo can temporarily remove their headsets withoutinterrupting their viewing of the story. In alternative, or in addition,the VR or AR headset may be equipped to easily switch between immersiveand non-immersive mode without interrupting play. However, suchswitching will not address all reasons why a user may wish to removeimmersive gear during a performance. Audience members may desire totemporarily remove their headsets for other reasons, such as to interactwith another audience member, enjoy a snack, adjust a fitting, relievethemselves of the head weight, cool their heads, or to walk to therestroom or concession stand, while following the action on the 2Dscreen.

Yet another benefit to theater owners is the ability to accommodatedifferent tiers of ticket prices within the same facility. Currently,for example, audiences paying higher ticket prices to enjoy stereoscopic3D content must be accommodated in a different theater than those payinglower prices to enjoy 2D content, requiring expensive duplication ofphysical facilities. In a coordinated 2D/AR/VR system, a theateroperator can implement tiered pricing for viewing content in the sameviewing room. For example, the operator may charge a basic ticket pricefor access to the theater, a separate fee for connecting a patron'sAR/VR equipment to the theater's immersive data stream, and anotherseparate fee for renting AR/VR equipment to patrons who do not bringtheir own equipment.

Referring to FIG. 4, a system 400 for providing immersive andnon-immersive content is illustrated in block form. Elements of thesystem 400 include a data source 408, coupled to a data distributionserver 422. In a cinema or home theater application, one or moreimmersive output devices, for example a VR output device 404 and an ARoutput device 406 (each being an example of an “immersive outputdevice”), are in communication with the server 422. Each immersiveoutput device may be coupled to a corresponding user input device 405,407. The user input devices 405, 407 may include one or more position,orientation, or motion sensors coupled to a user's body, and/or acontrol panel operable by user manipulation or other bodily input.Information derived from such sensors may be provided to components ofthe server 422. Contemporaneously with providing data streams to theimmersive output devices 404, 406, the server 422 may also provide adata stream to a projector 420 or other 2D display device, for examplean electronic display screen.

Data of various types may be obtained by the server 422 from the datasource 408. These types may include 2D data 412 for example a digitalvideo file or streaming feed, audio data (not shown), AR data 414 foraugmenting the 2D data using an AR output device 406, VR data 416 forproviding a parallel or supplemental entertainment experience using a VRoutput device 404, and a distribution application 418 for managingdistribution of the aforementioned data from the server 422. The server422 may execute the application 418, which when executed may providevarious functional modules. The modules may include a distributionmodule 424 for managing communication with and distribution to multipleimmersive output devices. The modules may include a decryption anddecoding module 426 for managing content security and providingstreaming data in a device-useable form. The modules may include acontent coordination module 428 for maintaining coordination betweenentertainment content streamed to different immersive output devices, acustomization module 430 for enabling content to be customized for aparticular immersive output device, for example in the case ofinteractive content. The modules may include a metrics module 432 forcollecting feedback from immersive output devices 404, 406, which may beanonymized and used to analyze use patterns with the aim of providingmore effective and compelling content for immersive output, for trackinguser preferences, or other purposes.

Narrative content represented by, for example, a motion picture script,may be produced for both immersive and non-immersive output devices.Referring to FIG. 5A, general aspects 500 of packaging immersive andnon-immersive content in a media package 502 are illustrated. The mediapackage 502 may be, or may include, a particular article, such ascomputer-readable optical disk or memory device. In the alternative, thepackage 502 may be, or may include, a set of data maintained on a serverfor which access rights are granted to a particular user account. Ineither case, the combining of immersive and non-immersive content asexemplified by the media package 502 is designed to appeal to a consumerdesiring to obtain access to immersive content 508 and non-immersivecontent 510 on different devices, whether at home or in a publictheater. For example, the consumer may desire to watch non-immersivecontent 510 on a video display screen of a mobile or larger device, andimmersive content 508 using a head set or other device that providesaccess to VR or AR content.

The non-immersive content 510 may be recorded according to anon-immersive narrative 506, for example, a traditional script. Theimmersive content 508 may be recorded according to an immersivenarrative ruleset 504, such as, for example, a branching narrative ofsome kind, or in the alternative, the same script as the non-immersivecontent. Both the immersive narrative ruleset 504 and the non-immersivenarrative 506 may be an expression of a narrative backbone. For example,the narrative backbone may include the entire narrative ruleset 504,while the non-immersive narrative 506 may be a subset of the backbone512, containing only selected critical narrative events arranged in aparticular narrative sequence.

In an aspect, immersive content 508 and non-immersive content 510 may becoordinated for consumption by playback in parallel. FIG. 5B illustratesaspects of parallel consumption 501 using coordinated output devices.The different output devices may obtain content from common source, forexample from a content server via a local area network or wireless localarea network in a cinema or home theater. A first output device fornon-immersive content may include a 2D display screen 526. A secondoutput device 528 (immersive output device) may be configured forproviding AR or VR. The different output devices 526, 528 may be in useby the same user 530, or by different users (not shown) occupying ashared space.

A data source may supply at least three types of data from a mediapackage: 2D or stereographic 3D frame data 516, VR or AR data 520, and amap 518 that relates the frame data 516 and the VR/AR data 520. Use ofthe map 518 may change as a function of screen geometry data 512 fromthe screen output controller 522, and geometry data defining the viewspace 514, for example a position and orientation of the viewer 530relative to the display screen 526 from sensors in the VR/AR outputdevice 528. The screen output controller 522 may play frame data in aconventional fashion for output on the display screen 526. While viewingthe output on the display screen 526, the user may also view output onthe VR/AR device 528. In a VR mode, the immersive output device 528 mayduplicate the view on the screen 526 and add additional surroundingimagery and interactive content. In an AR mode, the immersive outputdevice 528 may augment the display 526 with surrounding imagery orinteractive content. Using VR or AR content keyed to non-immersivecontent in a media package, a suitably equipped user can thus greatlyexpand the viewing area and number of interactive objects that can beexperienced in connection with the narrative content, relative to thecontent displayed on the display screen 526. The VR/AR output controller524 may keep the VR or AR output synchronized with play of the framedata 516, via the map 518 and geometry data 512, 514. The VR/AR outputcontroller 524 may also generate signals for controlling a haptic outputdevice 532, for example, an oscillator or pneumatically-activatedpressure reservoir.

Any of the features described herein may be executed by an applicationfor providing a 3D environment responsive to user input that produces VRoutput for an immersive VR headset or the like. FIG. 6 is a diagramillustrating one type of an immersive VR stereoscopic display device 600may be provided in various form factors, of which device 600 providesbut one example. The innovative methods, apparatus and systems are notnecessarily limited to a particular form factor of immersive VR displaybut may be used in a video immersive output device that enables the userto control a position or point of view of video content playing on thedevice. Likewise, a VR or AR output device may manage an audio positionor point of view of audio content playing on the device. The immersiveVR stereoscopic display device 600 represents an example of a relativelylow-cost device designed for consumer use.

The immersive VR stereoscopic display device 600 may include a tabletsupport structure made of an opaque lightweight structural material(e.g., a rigid polymer, aluminum or cardboard) configured for supportingand allowing for removable placement of a portable tablet computing orsmartphone device including a high-resolution display screen, forexample, an LCD display. This modular design may avoid the need fordedicated electronic components for video output, greatly reducing thecost. The device 600 is designed to be worn close to the user's face,enabling a wide field of view using a small screen size such astypically found in present handheld tablet computing or smartphonedevices. The support structure 626 may provide a fixed mounting for apair of lenses 622 held in relation to the display screen 612. Thelenses may be configured to enable the user to comfortably focus on thedisplay screen 612 which may be held approximately one to three inchesfrom the user's eyes.

The device 600 may further include a viewing shroud (not shown) coupledto the support structure 626 and configured of a soft, flexible or othersuitable opaque material for form fitting to the user's face andblocking outside light. The shroud may be configured to ensure that theonly visible light source to the user is the display screen 612,enhancing the immersive effect of using the device 600. A screen dividermay be used to separate the screen 612 into independently drivenstereoscopic regions, each of which is visible only through acorresponding one of the lenses 622. Hence, the immersive VRstereoscopic display device 600 may be used to provide stereoscopicdisplay output, providing a more realistic perception of 3D space forthe user. Two separate displays can also be used to provide independentimages to the user's left and right eyes respectively. It should beappreciated that the present technology may be used for, but is notnecessarily limited to, stereoscopic video output.

The immersive VR stereoscopic display device 600 may further comprise abridge (not shown) for positioning over the user's nose, to facilitateaccurate positioning of the lenses 622 with respect to the user's eyes.The device 600 may further comprise an elastic strap or band 624, orother headwear for fitting around the user's head and holding the device600 to the user's head.

The immersive VR stereoscopic display device 600 may include additionalelectronic components of a display and communications unit 602 (e.g., atablet computer or smartphone) in relation to a user's head 630. Asupport structure 604 holds the display and communications unit 602using restraining device 624 that is elastic and/or adjustable toprovide a comfortable and secure snug fit, for example, adjustableheadgear. When wearing the support 602, the user views the display 612though the pair of lenses 622. The display 612 may be driven by theCentral Processing Unit (CPU) 602 and/or Graphics Processing Unit (GPU)610 via an internal bus 616. Components of the display andcommunications unit 602 may further include, for example, atransmit/receive component or components 618, enabling wirelesscommunication between the CPU and an external server via a wirelesscoupling. The transmit/receive component 618 may operate using anysuitable high-bandwidth wireless technology or protocol, including, forexample, cellular telephone technologies such as 3rd GenerationPartnership Project (3GPP) Long Term Evolution (LTE), Global System forMobile communications (GSM) or Universal Mobile TelecommunicationsSystem (UMTS), and/or a wireless local area network (WLAN) technologyfor example using a protocol such as Institute of Electrical andElectronics Engineers (IEEE) 802.11. The transmit/receive component orcomponents 618 may enable streaming of video data to the display andcommunications unit 602 from a local or remote video server, and uplinktransmission of sensor and other data to the local or remote videoserver for control or audience response techniques as described herein.

Components of the display and communications unit 602 may furtherinclude, for example, one or more sensors 614 coupled to the CPU 606 viathe communications bus 616. Such sensors may include, for example, anaccelerometer/inclinometer array providing orientation data forindicating an orientation of the display and communications unit 602. Asthe display and communications unit 602 is fixed to the user's head 630,this data may also be calibrated to indicate an orientation of the head630. The one or more sensors 614 may further include, for example, aGlobal Positioning System (GPS) sensor indicating a geographic positionof the user. The one or more sensors 614 may further include, forexample, a camera or image sensor positioned to detect an orientation ofone or more of the user's eyes. In some embodiments, a cameras, imagesensor, or other sensor configured to detect a user's eyes or eyemovements may be mounted in the support structure 626 and coupled to theCPU 606 via the bus 616 and a serial bus port (not shown), for example,a Universal Serial Bus (USB) or other suitable communications port. Theone or more sensors 614 may further include, for example, aninterferometer positioned in the support structure 604 and configured toindicate a surface contour to the user's eyes. The one or more sensors614 may further include, for example, a microphone, array ormicrophones, or other audio input transducer for detecting spoken usercommands or verbal and non-verbal audible reactions to display output.The one or more sensors may include, for example, electrodes to senseheart rate, a temperature sensor configured for sensing skin or bodytemperature of the user, or other medical sensors for collectingbiofeedback data.

While FIG. 6 diagrams an example of a VR device, it should beappreciated that immersive AR devices may include similar electroniccomponents, except for the display output components. The processor andmemory of an AR device will also differ in the software that isoperated; VR software is configured to drive a VR output device, whileAR software is configured to drive an AR output device. A VR outputdevice may comprise a display screen such as, for example, aconventional LCD or OLED screen that is positioned in the VR headset soas to obscure the user's view of anything except what is displayed onthe display screen. In contrast, an AR output device does not obscurethe user's view of her surroundings.

Several types of AR output devices are commercially available or underdevelopment. On one class of devices, a display screen includes atwo-dimensional array of pixels that are either set to a first state inwhich the pixels are transparent or a second state in which the pixelsare opaque. The display screen is worn by the user and positioned suchthat groups of pixels set to the opaque state display a virtual image,while adjacent pixels transmit the external scene. This results in thevirtual image being overlain in the scene, from the user's point ofview. In another class of devices, a computer-generated image isprojected onto a semitransparent screen or lens that is worn in a mannersimilar to the VR headset described above. The screen or lens is made ofa one-way transmissive material that passes light from one side andreflects from the other direction in one or more layers. Light forforming the augmented reality image is projected from a DLP or the likefrom one or more emitters placed near the user inside of the headset. Ina third class of AR devices, one or more video cameras mounted on theheadset capture the external scene nearly as the user would see it werethe view not blocked by a fully opaque screen in the headset, andcombine video from the cameras with computer-generated content in realtime to create an augmented display. In a fourth type of device, a laserprojector or projector mounted to the user's head and directed to theuser's eyes projects images directly on the user's eyes, making theretinas the only screen on which the augmented content is displayed.Examples of devices for immersive and non-immersive AR output includethe Hololens™ device under development by Microsoft™ Corporation, GoogleGlass™ by Google Inc., Digital Lightfield™ by Magic Leap, Inc. ofFlorida; Space Glasses™ by Meta Company of Portola Valley, Calif., andcastAR™ glasses by of castAR or Palo Alto, Calif.

For immersive VR/AR or similar output modalities, the story content of amovie or the like may be enhanced, without eliminating the essence ofscripted entertainment that a participant or user (who is visually,aurally and cognitively immersed) can more or less passively enjoy. Forexample, allowing users to move the viewpoint to see items occluded inthe main view as a scene unfolds may enable such users to absorbdramatic details that enhance understanding of the plot, add emotionalimpact, foreshadow events to come, or otherwise enhance enjoyment of ascripted storyline. An example of foregoing is enhancing story tellingby user-selected depth of focus about feedback loops among aninteractive VR narrative (or whatever form the narrative takes), and atleast two sense modalities plus one cognitive item. These modalities maysupplement rather than replace conventional cinema viewing techniques,such that some patrons may view a conventional on-screen version of thesame feature, while other patrons who desire a more immersive experiencecan wear immersive headgear in the same theater as the conventionalpatrons and contemporaneously enjoy access to the supplemental immersivefeatures.

Sensor data from the one or more sensors may be processed locally by theCPU to control display output, and/or transmitted to a server forprocessing by the server in real time, or for non-real time processing.As used herein, “real time” refers to processing responsive to userinput that controls display output without any arbitrary delay; that is,that reacts as soon as technically feasible. “Non-real time” refers tobatch processing or other use of sensor data that is not used to provideimmediate control input for controlling the display, but that maycontrol the display after some arbitrary amount of delay.

Components of the display and communications unit 602 may furtherinclude, for example, an audio output transducer 620, for example aspeaker or piezoelectric transducer in the display and communicationsunit 602 or audio output port for headphones or other audio outputtransducer mounted in headgear 624 or the like. The audio output devicemay provide surround sound, multichannel audio, so-called ‘objectoriented audio’, or other audio track output accompanying a stereoscopicimmersive VR video display content. Components of the display andcommunications unit 602 may further include, for example, a memorydevice 608 coupled to the CPU 606 via a memory bus. The memory 608 maystore, for example, program instructions that when executed by theprocessor cause the apparatus 600 to perform operations as describedherein. The memory 608 may also store data, for example, audio-videodata in a library or buffered during streaming operations. Furtherdetails regarding generation and use of VR environments may be asdescribed in U.S. Provisional Patent Application Ser. No. 62/088,496,filed Dec. 5, 2014, which is incorporated herein in its entirety byreference.

FIG. 7 illustrates geometrical aspect of a display environment 700 forcoordinated immersive and non-immersive content, including a real orvirtual display screen 704 and a virtual envelope or shell 702 used as avirtual projection surface for rendering the background of a scene in away that blends smoothly with a projection surface. While shown as ahemispherical dome, it should be appreciated that the shell 702 may beprovided in a variety of shapes. Closed curves without sharp edges maybe suited for most shell geometries. FIG. 7 shows a bottom edge to theshell 702 for illustrative clarity, but it should be appreciated thattransitions between shell surfaces should generally be curved to avoidrendering artifacts. The shell 702 encloses a volume that may bereferred to herein as the “view space volume” or similar terminology, inwhich the action of the scene occurs. It should be appreciated, however,that the shell need not entirely surround the viewer. For example, ashell 702 may extend above or to the sides of a flat display screenwithout any curvature, may curve towards the audience but not all theway around the audience, or may be untextured and invisible in selectedareas.

In implementations using 100% rendered output, use of a shell 702 isoptional, because rendering may be based on a model with an infiniteextent. However, use of a textured shell may provide the advantage offaster render times and facilitate (by simplifying computations)rendering of the transition area around a 2D screen 704, for ARapplications. Background images may be rendered on the shell usingsimple “ambient” shading, which does not require any raytracing orraytracing approximation for determining surface appearance, except forcomputing a visible portion of surface from the rendered viewpoint.Instead, each pixel is rendered at a specified color and brightness“baked” into a 2D texture supplied for the geometry shell, based on anaggregate of the 2D texture's pixels that corresponds to a renderedpixel. The level of brightness or white balance of the baked texture maybe adjusted in a computationally efficient batch or real-time process tomatch screen characteristics of a particular theater. The effect may beas if the extent of the display screen were extended over the entiretextured portion of the shell 702. Selected portions of the shell 702may be left untextured and unrendered for any desired reason, forexample, for dramatic focus, to manage production costs, or for facilitysafety. It should be appreciated that a theater may include more thanone screen 704, if desired.

The shell 702 does not necessarily coincide with the interior of thetheater or room on which the screen 704 exists or in which the viewers706, 708 are situated. AR output devices are characterized by allowing auser to view her actual environment while overlying the view withobjects rendered to as to appear inside the actual environment. Where anAR object is rendered, the actual environment is obscured. Therefore, ifit is desired to create an illusion that transforms a real object into arendered object in the AR view, it is necessary to completely obscurethe real object with a rendered object. For example, if it is desired toreplace a wall of the theater with part of a geometry shell 702 on whicha scene background is rendered, the shell needs to obscure the entirewall. However, if the shell is positioned in the AR space beyond theactual wall, the rendered background may not cover the entire wallunless the shell is completely closed. Objects rendered in the viewspace volume may appear partly against the background, and partlyagainst real structures in the theater, detracting from the intendedillusion. If the shell is completely closed and rendered visible, theentire field of view of the user will be a rendered view, and thus, theeffect is that of VR and not AR, in the sense that “AR” is used in thepresent disclosure. Accordingly, for AR output, the shell may begenerated to fit inside the viewing room, to avoid diminishing therealism of the immersive AR experience by creating unintended effectssuch as objects seeming to pass through walls. As such, the shell 702should exist as an objective feature for AR viewers, meaning it isplaced based on the geometry of the viewing screen and room, such thateach viewer has a different perspective view of the shell 702 dependingon their respective positions in the theater.

For VR output, the shell may extend to any desired dimension and thescreen 704 may be virtual, functioning mainly as a sort of focal point706 or “home” defining a geometrical relationship between the viewer andthe intended narrative focal point of a scene. For VR output, the shellmay be objectively or subjectively placed, based on user or directorpreferences. When subjectively placed, each VR viewer may view the shell702 from the same apparent starting position, which optionally may bevaried individually for each used in response to user input. Thegeometry of the shell 702, and whether it is subjectively or objectivelyplaced, may vary from scene to scene based on the dramatic objectivesfor each scene. These factors would usually be static for the durationof particular scenes. Transitions between shells of different shapes,and transitions between objective or subjective viewpoints of an object,are permitted and may be abrupt or gradual. A gradual transition may beimplemented using an algorithm to generate a series of intermediateshapes or viewpoints bridging between the desired endpoints.

Each scene may have a static focal point 706, which in the case of aphysical screen would remain fixed from scene to scene. While the screen704 is shown as highly curved, it should be appreciated that actualphysical screens will usually have much less curvature, or no curvature.When a screen is flat or less curved, the shell 702 may be blended tomatch the curvature or flatness of the physical screen around its edges,if necessary. The shape of the screen 704 should be expected to varyfrom theater to theater. To enable use of the same content with screensand theaters of different sizes, a custom or semi-custom (meaningselected from a range of standard shapes and sizes) shell may beselected for each theater, and the custom or semi-custom shell excludingthe screen 704 area textured and rendered at runtime based on abackground texture file for the shell and the field of view of eachviewer. For example, for each viewer, the server may select and transmita portion of the background texture based on the current viewer field ofview, plus some amount of additional area beyond the current field ofview to accommodate an expected amount of head movement (which could bevery slight, or vary depending on the speed by which principal objectsare moving through the scene). Thus, each immersive output device neednot be supplied with the entire background texture for every frame, ifthe background is changing from frame to frame. If the background or aportion thereof is static over several frames, it may be lessresource-intensive to supply every immersive output device with theentire background texture or the static portion for the set of frames,instead of selecting a currently viewed portion for individual viewers.

In AR-coordinated viewings, the screen 704 essentially acts as a sort ofdynamic background that changes in virtually every frame, but thisportion of the output need not be supplied to the AR output device,because it is directly visible to the viewer. The surrounding backgroundportion may be static or dynamic, depending on the scene. Also, thesupplemental background placed on the shell 702 need not be supplied inevery scene, whether for dramatic effect, to manage production costs, orfor other reasons. For some scenes, the supplemental background may belimited to a relatively small portion adjacent to the screen 704 or maybe entirely omitted. In general, if supplemental content is to beprovided only for certain times of the performance, a visible or audiblecue may be provided in advance of each such time, to alert viewers toactivate their immersive output devices.

A geometrical relationship between the objective geometry for any givenscene and each viewer wearing AR or VR gear may be defined by a vector710, 712 from a fixed focal point 706 (e.g., a point at the center ofthe display screen) and each viewer, assuming that the viewer is gazingat the focal point so the view plane of the immersive device isperpendicular to each of the respective vectors 710, 712. Accordingly,to calibrate immersive output devices at the beginning of a showing, theaudience members wearing immersive gear may be instructed to gaze at afocal point 706 shown on screen, or several different points in turn,while each person's immersive head gear records a position andorientation of the gaze point. In addition, audience members may beinstructed to perform other movements while position and orientationmeasurements are similarly taken and recorded by each person's immersivehead gear. Optionally, individualized adjustments in brightness andwhite point may similarly be facilitated by measuring a white point andbrightness of one or more screen areas, using a light sensor on eachperson's immersive head gear. The recorded measurement data may then beused to calculate an individualized viewpoint location, orientation andlighting parameter adjustment for each audience member wearing immersivegear. The system server may then record the location and baseorientation for each viewer, or each immersive output device may recordits own location and orientation relative to the theater's objectivecoordinate system, or both.

As already noted, the position of off-screen rendered objects may bespecified using an objective coordinate system, for example a coordinatesystem having an origin at the focal point 706 or other location anddefined by set of coordinate axes from that origin. The flying dragon714 provides an example of an objectively located off-screen object, forwhich each user's perspective is different. For example, if theobjective coordinates for the dragon 714 indicate a position near thecenter of the view space volume, each user's immersive output devicewill output a rendering of the dragon positioned in the center of thetheater. A user 706 positioned towards the right of the theater will seethe dragon 714 to her left, while another user 708 positioned towardsthe left of the theater will see the dragon 714 to her right. Inaddition, off-screen objects may be specified relative to each user,based on respective subjective coordinate systems defined by each user'sequipment during a calibration sequence. The birds 716, 718 provideexamples of subjectively located off-screen objects, the position andorientation of which are specified in coordinates relative each viewer.Accordingly, both users will see their respective subjective off-screenobjects (e.g., birds 716, 718) in the same position and orientationrelative to themselves. It should be appreciated that user 706 will seebird 716 only, and not any other instance of the same subjective object,and likewise user 708 will see only her respective object 718. Whetheror not a set of object coordinates is subjective, or objective, may beindicated by a bit flag. For example, objective coordinates may beindicated by a ‘1’ value and subjective coordinates by a ‘0’ value ofthe flag bit, or vice-versa, for one or more coordinate sets.

Various adjustments to rendering parameters for immersive data may beapplied between the common “standard” content source (e.g., digitalcinematic master data) and the individualized rendering process by eachuser. FIG. 8A illustrates elements of a computer-implemented process 800for making such adjustments. Any one or all of the illustrated elementsof the process 800 may be individually performed by each user'simmersive output equipment, or (except for output of rendered immersivecontent) by a cinema server or network. Initially, at 802, immersivedata in digital cinematic master data is obtained from any suitable datasource and decrypted to obtain frame rendering parameters for each frameor set of frames. Such parameters may include, for example, shellobjects and off screen objects appearing in the scene, position andorientation of all objects to be rendered each associated with a set ofposition and orientation coordinates that are indicated as subjective orobjective, associated object textures for rendered objects, lightingparameters, and camera parameters. Standard frame rendering parametersmay then be adjusted for each frame or for sets of multiple contiguousframes, as necessary.

These adjustments may include, at 804, transforming objectivecoordinates for indicated objects to the coordinate system used by theapplicable render engine for rendering a viewpoint. Generally, thetransform 804 will transform an object's objective coordinates into thecoordinates used by the applicable render engine for rendering immersiveoutput for a particular immersive output device. The applicable renderengine may be located variously, such as in a random access memory of animmersive device, in a local auxiliary device for the immersive outputdevice, in a cinema server or server farm, or in a cloud computingresource. In any case, the coordinate transform will be based on thecoordinates used by the render engine and calibration data establishingthe geometrical relationship between each member of the audience and thetheater's objective coordinates. Any suitable transform method as knownin the art may be used for the coordinate transform.

The adjustments may further include, at 806, transforming subjectivecoordinates for indicated objects to the coordinate system used by theapplicable render engine for rendering a viewpoint. In the trivial case,no transformation is needed because the common subjective values willwork for every render engine and are the same for every audience member.However, in some cases certain transformation may be needed to putsubjective coordinates in proper condition for rendering, for exampleconverting to a different type of coordinate system to facilitate aparticular render engine or adding a fixed offset value to account forphysical differences between users.

The adjustments may further include, at 808, adjusting a position ororientation of rendered objects based on user input, in the case ofinteractive objects. The appearance, position, or orientation ofselected objects may depend on user input. The influence of user inputmay be limited to specified objects and ranges of change, to preventdisrupting the flow of a narrative performance and maintaincontemporaneous audience members in sync.

The adjustments may further include, at 810, adjusting scene lightingparameters. In an aspect, position and orientation of scene lights maybe designated objective or subjective and transformed as needed like anyother off screen object with respect to position and orientationcoordinated. In addition, other lighting parameters, such as intensityor color, may also be adjusted so that the brightness and color ofrendered scene elements matches the brightness and color of output onthe theater's display screen.

The adjustments may further include, at 812, adjusting object texture,for example, applying an automatic level of detail based on a distancebetween the rendered viewpoint and each rendered object, or equivalentmeasure. Automatic level of detail provides less detailed texture mapsfor more distant objects, to improve rendering performance. Similarly,automatic level of detail adjustments may be used to select a meshdensity for off screen objects based on distance from the viewpoint,again for rendering efficiency.

The adjustments may further include, at 814, adjusting camera parametersother than position and orientation, such as focal point, field offield, and aperture, based on immersive input. Hence, an immersiverender engine may allow a user to “zoom in” or “zoom out” on the scene,with appropriate camera adjustments. In AR mode with a common displayscreen, such camera zoom may cause mismatch between the display screen704 and rendered shell 702 or off screen objects 714, 716, 718, andmight be limited to rendering of views that exclude the display screenonly. Once adjustments are made the render engine may render the sceneat 816 and the rendered data may be displayed using an immersive outputdevice at block 818.

In a theater environment with seated patrons, the location of eachviewer may be relatively static, but the orientation of the viewer'shead will vary throughout the show. The cinema distribution server maytrack locations and/or orientation data for managing consumption ofbandwidth as mentioned above or may ignore position and orientationinformation if the theater's information system can support theprovision of full immersive environmental information to each user forevery frame. If the bandwidth of the information system is sufficientlybroad, and the computational power of the individual output devices issufficiently high to render all immersive features in every frame, allcomputation of the individual views may be performed at the respectiveimmersive output devices. Assuming present trends in computing power andtransmission bandwidth continue, it is likely that location andorientation tracking by a distribution server may at some point becomeunnecessary, unless for gathering statistical viewer metrics. In theinterim, computational power may need to be allocated to the serverside, for tasks such as managing bandwidth or providing high speed, highquality rendering in real time.

In another alternative embodiment 850 illustrated by FIG. 8B, animmersive and non-immersive content may be downloaded or otherwiseprovided to a plurality of dedicated memory devices 858A, 858B, each ofwhich is attached to a respective one of the immersive output devices856A, 856B via a high-bandwidth connection, such as, for example, ahigh-speed Universal Serial Bus (USB) or Serial Advanced TechnologyAttachment (SATA) connection. Each of the memory devices 858A, 858B maybe incorporated into the immersive output devices 856A, 856B, orinstalled in the theater and connected to the immersive output devices856A, 858B via a cable port at each chair and a corresponding one of therender control units 852A, 852B. The render units 852A, 852B may each beimplemented to optimize efficiency and comfort depending on availabletechnology, for example in a random access memory or specializedgraphics hardware subsystem of an immersive device, in a locally sitedauxiliary hardware and software component for the immersive outputdevice (e.g., under a cinema seat), in a cinema server or server farm,or in a cloud computing resource. Each one of the render control units852A, 852B may obtain the content from a respective one the memorydevices 858A, 858B based on sensor data from a corresponding one of theimmersive output devices 856A, 856B. The pace of the performance for thecinema system 850 may be controlled and coordinated between differentaudience members and a 2D display screen 870 by a signal from a cinemaserver 872, for example a wireless signal, generated by a coordinationmodule 878. In these embodiments, content data is provided to eachimmersive output device via respective high-bandwidth bus connectionswhile control signals used for coordinating the program over all theaudience can be provided in real time over a lower bandwidth wired orwireless interface. The content data 864, 866, 862 may be stored inencrypted form subject to a security system module 880 that only allowsaccess during the theater presentation or at other authorized times. ARdata 864 may be rendered for the AR output device 854A and VR data 866may be rendered for the VR output device 854B. Sensor data from userinput devices 856A, 856B may primarily be used to control rendering bythe rendering units 852A, 852B and secondarily may be provided to ametrics component 882 for tracking aggregate audience reactions toimmersive content. A communications software component 876 may be usedto manage communications between render units 852A, 852B and server 872,immersive output devices 854A, 854B and other system components.Likewise, an administrative software component 868 may manage overallsystem operation at server 872.

Although relieving bandwidth constraints on providing content data,providing the content over a bus may solve only half of the resourceproblem. Present graphics processors for rendering cinematic-quality,high resolution images in real time may have power consumption and spacerequirements that make them unsuitable for incorporating into wearable,mobile computing equipment. Hence, cinema-side implementations with highbandwidth connections, for example render management units 852A, 852Bimplemented local to cinema seating may offer a feasible solution to theintensive rendering demands of high-quality immersive content, in thenear term. As graphics rendering technology improves, these units may beupgraded periodically to more capable processors with lower powerrequirements, and eventually phased out when mobile, battery poweredprocessors are powerful enough to handle the processing load. In thealternative, or in addition, very high bandwidth communicationstechnology and improved image compression technology may permitrendering to be offloaded to cloud-implemented rendering units.

In another aspect, immersive content may be used to supplementnon-immersive content in a non-contemporaneous performance method 900,as illustrated by FIG. 9. As indicated at block 902, 2D (orstereographic 3D) content may be played for projection or otherconventional 2D screen display, without accompanying immersive play. Asan immersive play segment approaches, at 904, some signal may beprovided to audience members that possess immersive output equipmentthat an immersive segment is imminent. For example, text or graphics maybe included in the screen display, or the audience immersive outputdevices automatically turned on. At 906, optionally the 2D play may bepaused or set to an “immersive intermission” mode. In an alternative,the 3D play may continue contemporaneously. At 908, the cinema systemmay enable and cause immersive output in any suitable manner, asdescribed herein. At 910, when the immersive segment is finished, thecinema system may disable immersive play and revert to the 2Dperformance only at 902.

In view the foregoing, and by way of additional example, FIGS. 10-12show aspects of a method or methods for providing contemporaneousimmersive and non-immersive play in a cinema or home theater, as may beperformed by a home or cinema server or other computing apparatusdescribed herein. Referring to FIG. 10, a computer-implemented method1000 may include, at 1010, communicating with multiple immersive outputdevices each configured for providing one of an augmented reality (AR)output or a virtual reality (VR) output based on a data signal, whereineach of the multiple immersive output devices is present within eyesightof a 2D display screen located in a room, such as in a home theater orcommercial cinema. The communicating may be performed by any one or moreof a cinematic data distribution server, a home distribution server, ora network of dedicated immersive rendering units. The method mayinclude, at 1020, configuring the data signal based on digital cinematicmaster data that includes at least one of VR data or AR data. The methodmay include, at 1030, transmitting the data signal to the multipleimmersive output devices contemporaneously with each other, while in theeyesight of the display screen.

The method 1000 may include any one or more of additional operations1100, 1200 or 1300, shown in FIGS. 11, 12 and 13, in any operable order.Each of these additional operations is not necessarily performed inevery embodiment of the method, and the presence of any one of theoperations 1100, 1200 or 1300 does not necessarily require that anyother of these additional operations also be performed.

Referring to FIG. 11, the method 1000 may further include, at 1110,outputting an image based on a video data portion of the digitalcinematic master data on the display screen, contemporaneously with thetransmitting. As noted in connection with FIG. 9, display on the 2Ddisplay screen may also be performed non-contemporaneously withimmersive output.

In another aspect, the method 1000 may further include, at 1120,including the AR data in the data signal, augmenting video data outputon the display screen. Augmenting may include, at 1130, configuring thedata signal so when received by the multiple immersive output devices,images on the display screen are continuously extended to areas beyondan outer limit of the display screen, for each person viewing immersiveoutput on one of the multiple immersive output devices. In anotheraspect, augmenting may include, at 1140, including in the AR data code(e.g., program instructions or parameters) for managing a transitionbetween AR output produced from the AR data by an AR output device andvideo output shown on the display screen, based at least in part on ageometry of the display screen and a relative position of each personviewing AR output on one of the multiple immersive output devices.

Referring to FIG. 12, the method 1000 may further include, at 1210,configuring the data signal so that when received by the multipleimmersive output devices, images that do not appear on the displayscreen are caused to appear as if located in or around a view spacevolume to each person viewing immersive output on one of the multipleimmersive output devices. For example, the immersive data may cause anoff screen object or background shell to be rendered and output, asdescribed in connection with FIG. 7. In another aspect, the method 1000may further include, at 1220, configuring the data signal so that theimages that do not appear on the display screen appear in a coordinatesystem defined relative to the each person viewing immersive output onone of the multiple immersive output devices (i.e., in subjectivecoordinates as described in connection with FIG. 7). In addition, or inthe alternative, the method 1000 may further include, at 1230,configuring the data signal so that the images that do not appear on thedisplay screen appear in a coordinate system defined relative to thedisplay screen (i.e., in objective coordinates as described inconnection with FIG. 7).

In another aspect, the method 1000 may further include, at 1240,including in the data signal code enabling each person viewing immersiveoutput on one of the multiple immersive output devices to interact withat least one of the images that do not appear on the display screen,causing immersive output to change, as further described in connectionwith FIG. 8A. In an alternative, or in addition, the method may include,at 1250, changing video output shown on the display screen based on theeach person's interaction with at least one of the images. For example,conditional video sequences may be prepared for certain scenes to beplayed on the 2D screen, and the sequence to be shown on the 2D theaterscreen selected by the cinema server based on the interactions of one ormore audience members with an immersive, off-screen object. Thus,audience members may enjoy a sensation of influencing not only their ownimmersive experience, but also the non-immersive shared experience.

In another aspect, the method 1000 may further include, at 1260,providing an electronic record of a license right to digital content(e.g., a digital certificate) to a user account data store associatedwith the each person, based the each person's interaction with at leastone of the images. Thus, audience members may enjoy the sensation ofearning a right to use digital content in other applications (forexample, in a video game or social media application) based oninteractions with an off-screen immersive object. For example, ifwatching an immersive cinema presentation of Lord of the Rings, anaudience member who possesses a license to a Lord of the Rings videogame may be provided with opportunities to “pick up” certain off-screenimmersive objects useful in the video game, while watching the immersivepresentation. If, for example, the audience member interacts with anoff-screen immersive weapon by “picking it up” or the like, theinteraction may be recorded and transmitted to a game server for theLord of the Rings game, which then releases an in-game representation ofthe weapon in the member's game account. When the user returns home andlogs into the game server, the weapon is there and ready to be used ingame play. In another aspect, the data store may be or include acomputer memory in an AR or VR output device worn by the audience member(i.e., “the each person”). In these embodiments, no transmission to aremote game server is required. The acquired license may be immediatelyused in a game that operates on the VR or AR output device or in anycomputer authorized to connect to the VR or AR output device and accessthe certificate of license right stored there.

Referring to FIG. 13, the method 1000 may further include, at 1310,including the VR data for augmenting video data for output on thedisplay screen. In a related aspect, the method 1000 may include, at1320, playing the video data and the VR data at different,non-overlapping times, such that ones of the multiple persons wearing aVR output device view content on the display screen and on a VR outputdevice at different times. In addition, or in the alternative, the videodata and the VR data are configured to play contemporaneously.

In another aspect, the method 1000 may include, at 1340, includinghaptic control data in the digital cinematic master data, where thehaptic control data activates a haptic device in coordination with theat least one of AR data or VR data. Further details may be as describedin connection with FIG. 5B above.

FIG. 14 is a conceptual block diagram illustrating components of anapparatus or system 1400 for providing contemporaneous immersive andnon-immersive play in a cinema or home theater, as described herein. Theapparatus or system 1400 may include additional or more detailedcomponents for performing functions or process operations as describedherein. For example, the processor 1410 and memory 1416 may contain aninstantiation of an immersive content transformation process asdescribed herein above. As depicted, the apparatus or system 1400 mayinclude functional blocks that can represent functions implemented by aprocessor, software, or combination thereof (e.g., firmware).

As illustrated in FIG. 14, the apparatus or system 1400 may comprise anelectrical component 1402 for communicating with multiple immersiveoutput devices each configured for providing one of an AR output or a VRoutput based on a data signal, wherein each of the multiple immersiveoutput devices is present within eyesight of a display screen. Thecomponent 1402 may be, or may include, a means for the communicating.Said means may include the processor 1410 coupled to the memory 1416,the receiver 1414, and to the transmitter 1414, the processor executingan algorithm based on program instructions stored in the memory. Suchalgorithm may include a sequence of more detailed operations, forexample, establishing a plurality on concurrent data sessions with acorresponding plurality of the multiple immersive output devices,authenticating the multiple immersive output devices, establishing aconcurrent data session with a 2D display device, authenticating the 2Ddisplay device, and confirming that a content package for a performanceis authorized for play by the multiple immersive output devices and bythe 2D display device. Said means may also include, or in thealternative include, components of the systems 400, 850 described abovein connection with FIGS. 4 and 8B.

The apparatus 1400 may further include an electrical component 1404 forconfiguring the data signal based on digital cinematic master data thatincludes at least one of VR data or AR data. The component 1404 may be,or may include, a means for configuring the data signal based on digitalcinematic master data that includes at least one of VR data or AR data.Said means may include the processor 1410 coupled to the memory 1416,the processor executing an algorithm based on program instructionsstored in the memory. Such algorithm may include a sequence of moredetailed operations, for example, any one or more of the operationsdescribed in connection with FIGS. 11-13 that relate to configuring adata signal.

The apparatus 1400 may further include an electrical component 1406 fortransmitting the data signal to the multiple immersive output devicescontemporaneously. The component 1406 may be, or may include, a meansfor transmitting the data signal to the multiple immersive outputdevices contemporaneously. Said means may include the processor 1410coupled to the memory 1416, the receiver 1414, and to the transmitter1414, the processor executing an algorithm based on program instructionsstored in the memory. Such algorithm may include a sequence of moredetailed operations, for example, selecting a transmission mediumincluding at least a wireless medium or a wired medium, selecting acommunication protocol for the transmission medium, and transmitting thedata signal according to the communication protocol.

The apparatus 1400 may optionally include a processor module 1410 havingat least one processor, in the case of the apparatus 1400 configured asa data processor. The processor 1410, in such case, may be in operativecommunication with the modules 1402-1406 via a bus 1412 or similarcommunication coupling. The processor 1410 may effect initiation andscheduling of the processes or functions performed by electricalcomponents 1402-1406.

In related aspects, the apparatus 1400 may include a network interfacemodule (not shown) operable for communicating with system componentsover a computer network. In further related aspects, the apparatus 1400may optionally include a module for storing information, such as, forexample, a memory device 1416. The computer readable medium or thememory module 1416 may be operatively coupled to the other components ofthe apparatus 1400 via the bus 1412 or the like. The memory module 1416may be adapted to store computer readable instructions and data foreffecting the processes and behavior of the modules 1402-1406, andsubcomponents thereof, or the processor 1410, or the method 1000 and oneor more of the additional operations 1100, 1200 or 1300 disclosedherein. The memory module 1416 may retain instructions for executingfunctions associated with the modules 1402-1406. While shown as beingexternal to the memory 1416, it is to be understood that the modules1402-1406 can exist within the memory 1416.

The apparatus 1400 may include a transmitter 1412 configured as awireless transmitter, or a wired transmitter, for transmitting acommunication signal to a VR output device, an AR output device, oranother system component such as, for example, the rendering units shownin FIG. 8B or the cinema server shown in FIGS. 4 and 8B. In addition,the apparatus 1400 may include a receiver 1414 for receiving acommunication signal from a VR output device, an AR output device, oranother system component such as, for example, the rendering units shownin FIG. 8B or the cinema server shown in FIGS. 4 and 8B. The receiver1414 may comprise a wireless receiver, or a wired receiver.

In view the foregoing, and by way of additional example with respect topreparing digital master data for immersive and non-immersivecontemporaneous performance, FIGS. 15-17 show aspects of a method ormethods for configuring digital cinematic master data, as may beperformed by an production device for VR or AR content or othercomputing apparatus described herein. Referring to FIG. 15, a method1400 may include, at 1410, configuring, by at least one computer,digital cinematic master data that includes at least one of augmentedreality (AR) data or virtual reality (VR) data for providing one of anAR output or a VR output comprising a cinematic presentationcontemporaneously with output for display on a 2D screen. The method1500 may further include, at 1520, recording the digital cinematicmaster data in a non-transitory computer-readable medium. Furtherdetails of the method 1500 may be consistent with the description abovein connection with FIGS. 1 and 4-9.

The method 1400 may include any one or more of additional operations1600 or 1700, shown in FIGS. 16 and 17, in any operable order. Each ofthese additional operations is not necessarily performed in everyembodiment of the method, and the presence of any one of the operations1600 or 1700 does not necessarily require that any other of theseadditional operations also be performed.

Referring to FIG. 16, the method 1500 may further include, at 1610,preparing the AR data for augmenting video data configured forprojection or output on the 2D display screen arranged for viewing bymultiple individuals. The data for output on the 2D screen may beconfigured for one or more of stereoscopic 3D output, or 2D output. Themethod 1500 may further include, at 1620, preparing the AR data forcontinuously extending images on the display screen to areas beyond anouter limit of the display screen. Examples of continuous extension aredescribed above in connection with FIGS. 3 and 7. The method 1500 mayfurther include, at 1630, preparing the AR data for causing images thatdo not appear on the display screen to appear as if located in or arounda non-screen display volume to any one of the multiple individualswearing an AR output device. For example, the data may be prepared fordisplay of off screen objects or geometry shells as described inconnection with FIGS. 3 and 7. The method 1500 may further include, at1640, preparing the AR data so that the images appear in a coordinatesystem defined relative to each of the multiple individuals wearing anAR output device (i.e., in a subjective coordinate system). The method1500 may further include, at 1650, preparing the AR data so that theimages appear in a coordinate system defined relative to the displayscreen (i.e., in an objective coordinate system).

Referring to FIG. 17, the method 1500 may further include, at 1710,including in the AR data code for enabling a user to interact with atleast one of the images, causing the AR output to change in response touser input. The method 1500 may further include, at 1720, configuringthe code so that a user's interaction with at least one of the imagescauses the video output shown on the display screen to change. Themethod 1500 may further include, at 1730, configuring the code so that auser's interaction with at least one of the images causes an electronicrecord of a license right to digital content to be provided to a useraccount data store associated with the user. The method 1500 may furtherinclude, at 1730, including haptic control data in the digital cinematicmaster data, where the haptic control data activates a haptic device incoordination with the at least one of AR data or VR data. In otheraspects, configuring the cinematic master data may further includeperforming operations to implement or facilitate any of the moredetailed output effects or associated functions described herein abovein connection with use of cinematic master data in contemporaneousimmersive and non-immersive performances for audiences of multiplepersons.

FIG. 18 is a conceptual block diagram illustrating components of anapparatus or system 1800 for preparing digital master data for immersiveand non-immersive contemporaneous performance as described herein. Theapparatus or system 1800 may include additional or more detailedcomponents as described herein. As depicted, the apparatus or system1800 may include functional blocks that can represent functionsimplemented by a processor, software, or combination thereof (e.g.,firmware).

As illustrated in FIG. 18, the apparatus or system 1800 may comprise anelectrical component 1802 for configuring digital cinematic master datathat includes at least one of augmented reality (AR) data or virtualreality (VR) data for providing one of an AR output or a VR outputcomprising a cinematic presentation contemporaneously with output fordisplay on a 2D screen. The component 1802 may be, or may include, ameans for configuring VR output data. Said means may include theprocessor 1810 coupled to the memory 1816, the processor executing analgorithm based on program instructions stored in the memory. Suchalgorithm may include a sequence of more detailed operations, forexample, arranging immersive content, arranging non-immersive content,and defining a set of rules for coordinating output of immersive contentby an immersive output device with contemporaneous output ofnon-immersive content for display on a 2D screen.

The apparatus 1800 may further include an electrical component 1804 forrecording the digital cinematic master data in a non-transitorycomputer-readable medium. The component 1804 may be, or may include, ameans for recording the digital cinematic master data in anon-transitory computer-readable medium. Said means may include theprocessor 1810 coupled to the memory 1816, the processor executing analgorithm based on program instructions stored in the memory. Suchalgorithm may include a sequence of more detailed operations, forexample, selecting an output format for the master data, formatting themaster data according to the selected output format, encrypting theformatted master data, and encoding the encrypted and formatted masterdata in a non-transitory computer-readable medium according to aninformation storage protocol.

The apparatus 1800 may optionally include a processor module 1810 havingat least one processor, in the case of the apparatus 1800 configured asa data processor. The processor 1810, in such case, may be in operativecommunication with the modules 1802-1804 via a bus 1812 or similarcommunication coupling. The processor 1810 may effect initiation andscheduling of the processes or functions performed by electricalcomponents 1802-1804.

In related aspects, the apparatus 1800 may include a network interfacemodule 1814 operable for communicating with system components over acomputer network. For example, the component 1814 may facilitatedistribution of the digital master data to other devices of acommunications and/or computer network. In alternative embodiments, theapparatus may include a port to a device for recording the content andalternative content in a non-transitory computer-readable medium. Infurther related aspects, the apparatus 1800 may optionally include amodule for storing information, such as, for example, a memorydevice/module 1816. The computer readable medium or the memory module1816 may be operatively coupled to the other components of the apparatus1800 via the bus 1812 or the like. The memory module 1816 may be adaptedto store computer readable instructions and data for effecting theprocesses and behavior of the modules 1802-1804, and subcomponentsthereof, or the processor 1810, or of the method 1500 and one or more ofthe additional operations 1600 or 1700 disclosed herein, or ofoperations to implement or facilitate any of the more detailed outputeffects or associated functions described herein above in connectionwith use of cinematic master data in contemporaneous immersive andnon-immersive performances for audiences of multiple persons. The memorymodule 1816 may retain instructions for executing functions associatedwith the modules 1802-1804. While shown as being external to the memory1816, it is to be understood that the modules 1802-1804 can exist withinthe memory 1816.

Further aspects of mastering for AR and VR are outlined in the attachedappendix.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the aspects disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

As used in this application, the terms “component”, “module”, “system”,and the like are intended to refer to a computer-related entity, eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component or a module may be, but are notlimited to being, a process running on a processor, a processor, anobject, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component or a module. One or morecomponents or modules may reside within a process and/or thread ofexecution and a component or module may be localized on one computerand/or distributed between two or more computers.

Various aspects will be presented in terms of systems that may include anumber of components, modules, and the like. It is to be understood andappreciated that the various systems may include additional components,modules, etc. and/or may not include all of the components, modules,etc. discussed in connection with the figures. A combination of theseapproaches may also be used. The various aspects disclosed herein can beperformed on electrical devices including devices that utilize touchscreen display technologies, heads-up user interfaces, wearableinterfaces, and/or mouse-and-keyboard type interfaces. Examples of suchdevices include VR output devices (e.g., VR headsets), AR output devices(e.g., AR headsets), computers (desktop and mobile), smart phones,personal digital assistants (PDAs), and other electronic devices bothwired and wireless.

In addition, the various illustrative logical blocks, modules, andcircuits described in connection with the aspects disclosed herein maybe implemented or performed with a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general purpose processor may be a microprocessor,but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

Operational aspects disclosed herein may be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, DVD, Blu-ray disc, solid-state storagedevice (SSD) or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.The processor and the storage medium may reside in an ASIC. The ASIC mayreside in a user terminal. In the alternative, the processor and thestorage medium may reside as discrete components in a user terminal.

Furthermore, the one or more versions may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedaspects. Non-transitory computer readable media can include but are notlimited to magnetic storage devices (e.g., hard disk, floppy disk,magnetic strips . . . ), optical disks (e.g., compact disk (CD), digitalversatile disk (DVD), Blu-ray™, . . . ), smart cards, flash memorydevices (e.g., card, stick), and solid-state storage devices. Of course,those skilled in the art will recognize many modifications may be madeto this configuration without departing from the scope of the disclosedaspects.

The previous description of the disclosed aspects is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments without departing from the spirit or scopeof the disclosure. Thus, the present disclosure is not intended to belimited to the embodiments shown herein but is to be accorded the widestscope consistent with the principles and novel features disclosedherein.

In view of the exemplary systems described supra, methodologies that maybe implemented in accordance with the disclosed subject matter have beendescribed with reference to several flow diagrams. While for purposes ofsimplicity of explanation, the methodologies are shown and described asa series of blocks, it is to be understood and appreciated that theclaimed subject matter is not limited by the order of the blocks, assome blocks may occur in different orders and/or concurrently with otherblocks from what is depicted and described herein. Moreover, not allillustrated blocks may be required to implement the methodologiesdescribed herein. Additionally, it should be further appreciated thatthe methodologies disclosed herein are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethodologies to computers.

1-20. (canceled)
 21. A method comprising: configuring, by at least oneprocessor in response to user input, digital cinematic master data thatincludes at least one of augmented reality (AR) data or virtual reality(VR) data for providing one of an AR output or a VR output comprising acinematic presentation contemporaneously with output for display on a 2Dscreen; wherein the digital cinematic master data enables locatingoff-screen objects coordinated with a cinematic presentation on adisplay screen shared by multiple viewers, so that one of the off-screenobjects appears in subjective viewpoints each relative to a differentone of the multiple immersive output devices and another of theoff-screen objects appears in an objective viewpoint relative to theshared display screen; and recording, by the at least one processor, thedigital cinematic master data in a non-transitory computer-readablemedium.
 22. The method of claim 21, wherein the configuring comprisespreparing the AR data for augmenting video data configured forprojection or output on the 2D display screen arranged for viewing bymultiple individuals.
 23. The method of claim 22, wherein theconfiguring comprises preparing the AR data for continuously extendingimages on the display screen to areas beyond an outer limit of thedisplay screen.
 24. The method of claim 21, wherein the configuringcomprises preparing the AR data for causing images that do not appear onthe display screen to appear as if located in or around a non-screendisplay volume to any one of the multiple individuals wearing an ARoutput device.
 25. The method of claim 24, wherein the configuringcomprises preparing the AR data so that at least one of the imagesappears in a coordinate system defined relative to each of the multipleindividuals wearing an AR output device.
 26. The method of claim 24,wherein the configuring comprises preparing the AR data so that at leastone of the images appears in a coordinate system defined relative to thedisplay screen.
 27. The method of claim 24, further comprising includingin the AR data code for enabling a user to interact with at least one ofthe images, causing the AR output to change in response to user input.28. The method of claim 27, further comprising configuring the code sothat a user's interaction with at least one of the images causes thevideo output shown on the display screen to change.
 29. The method ofclaim 27, further comprising configuring the code so that a user'sinteraction with at least one of the images causes an electronic recordof a license right to digital content to be provided to a user accountdata store associated with the user.
 30. The method of claim 29, wherethe data store comprises a computer memory in an AR output device wornby the user.
 31. The method of claim 22, further comprising including inthe AR data code for managing a transition between AR output producedfrom the AR data by an AR output device and video output shown on thedisplay screen, based at least in part on a geometry of the displayscreen and a relative position of a user wearing the AR output device.32. The method of claim 21, wherein the configuring comprises preparingthe VR data for augmenting video data configured for projection oroutput on a display screen arranged for viewing by multiple individuals.33. The method of claim 32, wherein the video data and the VR data areconfigured to play at different, non-overlapping times, such that onesof the multiple individuals wearing a VR output device view content onthe display screen and on a VR output device at different times.
 34. Themethod of claim 32, wherein the video data and the VR data areconfigured to play contemporaneously.
 35. The method of claim 21,wherein the video data comprises one of two-dimensional non-stereoscopicor stereoscopic frame data.
 36. The method of claim 21, furthercomprising including haptic control data in the digital cinematic masterdata, where the haptic control data activates a haptic device incoordination with the at least one of AR data or VR data.
 37. Anapparatus for configuring digital cinematic master data for virtualreality (VR) or augmented reality (AR) output, comprising: a processor,a memory coupled to the processor, and user interface device coupled tothe processor, wherein the memory holds instructions that when executedby the processor, cause the apparatus to perform: configuring, by atleast one processor in response to user input, digital cinematic masterdata that includes at least one of augmented reality (AR) data orvirtual reality (VR) data for providing one of an AR output or a VRoutput comprising a cinematic presentation contemporaneously with outputfor display on a 2D screen; wherein the digital cinematic master dataenables locating off-screen objects coordinated with a cinematicpresentation on a display screen shared by multiple viewers, so that oneof the off-screen objects appears in subjective viewpoints each relativeto a different one of the multiple immersive output devices and anotherof the off-screen objects appears in an objective viewpoint relative tothe shared display screen; and recording, by the at least one processor,the digital cinematic master data in a non-transitory computer-readablemedium.
 38. The apparatus of claim 37, wherein the memory holds furtherinstructions for configuring the video data and the VR data to play atdifferent, non-overlapping times, such that ones of the multipleindividuals wearing a VR output device view content on the displayscreen and on a VR output device at different times.
 39. The apparatusof claim 37, wherein the memory holds further instructions forconfiguring the video data and the VR data to play contemporaneously.40. The apparatus of claim 37, wherein the memory holds furtherinstructions for configuring the video data as one of two-dimensionalnon-stereoscopic or stereoscopic frame data.